Physical activity quantification and monitoring

ABSTRACT

Certain aspects provide a method of generating a physical activity model, including: receiving, via a motion capture device, motion data corresponding to a plurality of key states associated with a physical activity sequence; for each respective key state in the plurality of key states: determining a plurality of joint positions associated with the respective key state; determining a plurality of body segment positions associated with the respective key state based on the plurality of joint positions; determining a plurality of inter-state differentiation variables for the respective key state; determining one or more state characteristic metrics for the respective key state; and determining a classifier for the respective key state based on the one or more state characteristic metrics; and defining a physical activity model based on the one or more state characteristic metrics and the classifier associated with each key state.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent applicationSer. No. 16/653,153, filed Oct. 15, 2019, which application claims thebenefit of U.S. Provisional Patent Application No. 62/768,012, filed onNov. 15, 2018, each of which application is incorporated herein byreference in its entirety.

INTRODUCTION

Aspects of the present disclosure relate to systems and methods forquantifying and monitoring physical activities based on motion data, andin particular, to generating physical activity models based on motiondata captured from motion tracking systems.

In a physical rehabilitation setting, patients are often prescribedphysical therapies, which may include specific, physical activities,such as exercises targeting specific movements of specific limbs andjoints. Typically, a patient is given written instructions for when andhow to perform the physical activities (e.g., a certain number ofrepetitions of a specific exercise every twelve hours). Historically,patients have also needed to go to a physical therapy clinic, or to havea clinician visit them at home, to monitor their physical therapy and toget feedback and coaching to maximize compliance with their physicaltherapy. Such on-site monitoring generally improves the efficacy of thephysical therapy, for example, by ensuring that it is performedcorrectly and consistently. However, this conventional practice istime-consuming, expensive, and logistically challenging. Moreoverin-clinic or in-home physical therapy coaching may not be available tothose with limited mobility or means. While patients can performprescribed physical therapies on their own without professional support,there is no guarantee that the patients will follow instructions and useproper form—which is critical to the efficacy of the prescribed physicaltherapies. In fact, unsupported physical therapy frequently leads toinferior patient outcomes, higher chances of re-injury, and the like.

Notably, the same issues faced in the physical therapy context arepresent in other contexts, such as physical fitness training forperformance improvement rather than injury recovery, in coaching ofathletes for various sports, and in any other context where theconsistency and quality of body motions may improve a desired outcome.

Accordingly, what are needed are systems and methods for quantitativelydefining physical activities, which allow for automated monitoring andfeedback without the need for on-site personnel.

BRIEF SUMMARY

Certain embodiments provide a method of generating a physical activitymodel, comprising: receiving, via a motion capture device, motion datacorresponding to a plurality of key states associated with a physicalactivity sequence; for each respective key state in the plurality of keystates: determining a plurality of joint positions associated with therespective key state; determining a plurality of body segment positionsassociated with the respective key state based on the plurality of jointpositions associated with the respective key state; and determining aplurality of inter-state differentiation variables for the respectivekey state based on one or more of: the plurality of joint positionsassociated with the respective key state; or the plurality of bodysegment positions associated with the respective key state; determiningone or more state characteristic metrics for the respective key statebased on the plurality of inter-state differentiation variablesassociated with the respective key state; and determining a classifierfor the respective key state based on the one or more statecharacteristic metrics; and defining a physical activity model based onthe one or more state characteristic metrics and the classifierassociated with each key state.

Further embodiments provide a processing system, comprising: anon-transitory computer-readable medium comprising computer-executableinstructions; and a processor configured to execute thecomputer-executable instructions and cause the processing system toperform a method of generating a physical activity model, the methodcomprising: receiving, via a motion capture device, motion datacorresponding to a plurality of key states associated with a physicalactivity sequence; for each respective key state in the plurality of keystates: determining a plurality of joint positions associated with therespective key state; and determining a plurality of body segmentpositions associated with the respective key state based on theplurality of joint positions associated with the respective key state;determining a plurality of inter-state differentiation variables for therespective key state based on one or more of: the plurality of jointpositions associated with the respective key state; or the plurality ofbody segment positions associated with the respective key state;determining one or more state characteristic metrics for the respectivekey state based on the plurality of inter-state differentiationvariables associated with the respective key state; and determining aclassifier for the respective key state based on the one or more statecharacteristic metrics; and defining a physical activity model based onthe one or more state characteristic metrics and the classifierassociated with each key state.

Further embodiments provide a method for using a physical activitymodel, comprising: receiving motion data from a motion capture device;providing the received motion data to a physical activity model, whereinthe physical activity model comprises: a plurality of classifiers,wherein each classifier of the plurality of classifiers is associatedwith a key state of a physical activity; and a plurality of statecharacteristic metrics, wherein each state characteristic metrics of theplurality of state characteristic metrics is associated with one or moreof the plurality of classifiers; receiving, from the physical activitymodel, a plurality of scores, wherein each score of the plurality ofscores is associated with one of the plurality of classifiers; anddetermining a key state is represented in the received motion data basedon the plurality of scores.

Other embodiments include non-transitory computer-readable mediumscomprising computer-executable instructions for performing theaforementioned processes as well as the additional processes describedherein as well as processing systems configured to perform theaforementioned processes as well as the additional processes describedherein.

The following description and the related drawings set forth in detailcertain illustrative features of one or more embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The appended figures depict certain aspects of the one or moreembodiments and are therefore not to be considered limiting of the scopeof this disclosure.

FIG. 1 depicts an example flow for creating and using a physicalactivity model.

FIG. 2 depicts example output from a motion capture system showing thedifference between two physical activity states.

FIG. 3 depicts an example of determining candidate inter-statedifferentiation variables that may be used to distinguish between keystates of a physical activity.

FIG. 4 depicts an example of training a physical activity model based ona plurality of key states.

FIG. 5 depicts an example of determining physical activity stateprobabilities using a physical activity model.

FIG. 6 depicts an example of tracking a defined physical activity statesequence.

FIG. 7 depicts an example user interface for recording training data forphysical activity model development.

FIG. 8 depicts another view of an example user interface during atraining data recording process.

FIG. 9 depicts a portion of user interface in which various statecharacteristic metrics are displayed.

FIG. 10 depicts an example graphical user interface in an activitytracking mode.

FIG. 11 depicts another view of an example graphical user interface inan activity tracking mode.

FIG. 12 depicts another view of an example graphical user interface inan activity tracking mode.

FIG. 13 depicts an example method for training a physical activitymodel.

FIG. 14 depicts an example method 1400 for using a physical activitymodel.

FIG. 15 depicts an example processing system 1500 configured to generateand use physical activity models.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe drawings. It is contemplated that elements and features of oneembodiment may be beneficially incorporated in other embodiments withoutfurther recitation.

DETAILED DESCRIPTION

Aspects of the present disclosure provide apparatuses, methods,processing systems, and computer readable mediums for quantitativelydefining and monitoring physical activities based on captured motiondata.

Generally, a physical activity may include a bodily activity thatincludes a temporal sequence of states of a subject's body. Physicalactivities may take many different forms. For example, a physicalactivity may comprise an exercise or other prescribed motion performedor practiced in order to develop, improve, or display a physicalcapability or skill.

A state, or activity state, associated with a physical activity maygenerally include a particular position, pose, or bearing of thesubject's body, whether characteristic or assumed for a special purpose.One or more of the states in a sequence of states defining a physicalactivity may be considered a key state, which is a specific state thatin-part defines the physical activity. Determination of specific keystates is useful to ensure proper physical activity model formulationfor state differentiation.

For example, the physical activity of sitting down may have a first keystate of standing and a second key state of sitting. The states inbetween standing and sitting may not be considered “key” because theymay not be important to defining the overall physical activity ofsitting. Key states may be determined by experts, such as trainers,clinicians, doctors, or the like, or determined based on analyses oftemporal sequences of states associated with a physical activity. Forexample, points of inflection during the motion path of a particularbody segment or joint may indicate a key state of a particular physicalactivity.

In some cases, a physical activity may be further defined by temporalspecifications, such as a need to move from one state to another withina specified time, or a need to hold one state for a specified time, toname just a few examples.

Each state in a physical activity state sequence may be defined withreference to individual segments or portions of a subject's body, suchas the subject's head, neck, torso, arms, hands, fingers, legs, feet,bones, and others. Physical activities may be further defined by joints,which are generally connection points between two adjoining bodysegments that allow for some articulation of one segment in relation toanother connected segment. In some cases, the individual segments andjoints associated with a subject may be combined to form a digital bodyrepresentation, such as a skeleton representation, or other morefeatured representation, such as an avatar.

Physical activity state sequences may be captured and digitized througha process of motion capture, motion monitoring, motion tracking, or thelike, which all generally refer to a process for generating dataregarding a subject's kinematic motion and static poses using a varietyof electronic sensors. Generally, a motion tracking system may includehardware and software components configured to monitor a subject'sphysical activity (e.g., movements, exercises, etc.). In someembodiments, a motion capture device may include optical camera systemswith image processing, marker based tracking systems with various marker(active, passive, semi-passive, modulated) and detector (optical, radiofrequency) types, depth camera systems with object recognitionalgorithms, inertial measurement units, mechanical exoskeleton motioncapture systems, or magnetic flux measurement systems, to name a few.One example of a motion tracking system is the KINECT® sensor and itsassociated pose detection software by MICROSOFT®.

States in a physical activity state sequence may be compared to generatestate differentiation variables. For example, a plurality of candidateinter-state differentiation variables may be defined based on bodysegments and joints in order to identify or improve identification ofdifferences between states, including key states, of a physicalactivity. In some embodiments, the plurality of candidate inter-statedifferentiation variables may be tested to determine a subset ofinter-state differentiation variables that are most effective foridentifying particular states, such as key states, in captured motiondata. The selected subset inter-state differentiation variables may bereferred to as state characteristic metrics, which are generally used bya physical activity model for identification and tracking of key statesof a physical activity in motion data.

An ideal physical activity sequence may be defined, for example, bycapturing motion data of a professional performing a physical activityin a prescribed manner and thereafter defining key states of thephysical activity based on the captured motion data. As above, the keystates may be defined manually, such as by a professional, orautomatically, by analyzing the change in variables in captured motiondata during performance of the ideal physical activity sequence.Further, in some implementations, key states can be numerically definedwithout a sample of collected motion capture data, such as by use of askeletal data model.

Once a physical activity has been defined through quantification, asabove, a physical activity model may be generated to determine (e.g.,recognize or identify) states, such as key states, of the physicalactivity in captured motion data. Further, the physical activity modelmay compare the determined states with states of an ideal physicalactivity state sequence to “score” a subject's performance of thephysical activity and/or to provide live feedback to the subject on thequality of the performance of the physical activity. This enables, ineffect, live monitoring and feedback to a subject without the need foran on-site professional.

In some embodiments, a physical activity model comprises one or moreclassifiers that are configured to determine probabilities that aparticular state is represented in captured motion data. Further, foreach classifier of the physical activity model, a classifier confidencemay be determined as a quantitative assessment corresponding to theclassifier's performance in determining a correct state, classification,or category of captured motion data. Determination of particular statesvia the physical activity model may further lead to determination that adefined physical activity, which comprises some or all of the determinedstates, is represented in the captured motion data.

In some embodiments, a physical activity model further comprises one ormore state characteristic metrics, as described above. The combinationof classifiers and state characteristic metrics enables a singlephysical activity model to generate predictions regarding a plurality ofdefined states and physical activities in captured motion data.

Capturing Motion Data for Quantifying and Monitoring Physical Activities

In embodiments described herein, a physical activity monitoring systemis configured to capture motion data regarding a subject's physicalactivity (e.g., performing a specific movement or sequence of movementsas a part of an exercise), quantitatively compare the captured motiondata against one or more defined physical activity models, and toprovide real-time monitoring and feedback regarding the subject'sphysical activity. Notably, this may be done without the need for anon-site professional, such as a doctor, clinician, coach, trainer, orthe like. Even without a professional on-site, embodiments describedherein enable an off-site professional to, for example, review capturedmotion data, review monitoring data generated by one or more physicalactivity models, and review system generated feedback based on themonitoring data, to further improve feedback to the subject.

FIG. 1 depicts an example flow 100 for generating and using a physicalactivity model.

Flow 100 begins at step 102 with capturing motion data associated withkey states of a physical activity. As above, key states may includespecific body poses as well as specific temporal aspects of time in apose as well as time in transition between poses.

In some embodiments, the key states may be defined by a professional andperformed by a professional in order to quantitatively define thephysical activity through captured motion data. In other embodiments,the key states may be determined automatically based on captured motiondata, such as by identifying pauses, transitional sequences, or otherindicators that one key state is transitioning to another key state.

In one example, in a rehabilitation context, a trainer (e.g., an expertpractitioner, possibly a clinician) may predefine a sequence of keystates, including temporal characteristics, and then enact them in frontof a motion capture device. The motion capture device records thetrainer's joint positions as they perform the activity. Thus, key statescan be selected from a continuous execution of states of the activitywithout having the trainer hold a pose statically. In some embodiments,key states may be calculated based on a sequence of motion data in whichthe particular activity is repeated one or more times.

In some embodiments, the motion data captured by a motion capture systemmay be used to determine coordinates of a plurality of joint positions,from which body segments (e.g., limbs) can be determined to create askeleton model or reconstruction of the trainer's motion duringperformance of the physical activity. In some embodiments, each bodysegment is defined as a length between distinguishing joints, and a unitvector may further be defined which gives the orientation of the bodysegment in a coordinate frame. In some embodiments, joint angles mayalso be calculated based on comparing the unit vectors of adjoining bodysegments vectors using vector mathematics.

Herein, absolute and relative joint positions are described as one wayof tracking states of a subject's motions. However, the methodsdescribed herein are compatible with any mathematical representation ofa state, including body segments, joint positions, and other bodycharacteristics. Further, states may be represented by various methods,or combinations of methods, including: limb orientations, body spread,kinematic models of the body using rotation matrices or quaternions,body silhouette, body 3D point cloud spatial distribution metrics,parametric approaches, like the Sum of Gaussians (SoG) representation,and others. In other words, it is not necessary that a state berepresented only as a set of joint positions.

In some embodiments, a subject's joint positions are derived usingmathematical operations based on tracking data that does not directlyprovide joint positions. For example, a 3D point cloud system may notinherently provide joint positions, but machine learning approaches maybe used to get the joint positions from the 3D point cloud. Similarly,marker based systems may not always have markers at the joints, but mayuse kinematic models of the body to figure out the joint positions giventhe locations of the sensors.

In certain situations, it may be advantageous to record training data(at step 102) from multiple subjects and/or using multiple trackingmethodologies or devices. Differences can be found in the key states forsubjects of varying size and shape and depending on the motion capturedevice used. Diversifying the training data may beneficially providewider ranges of acceptable values for pose variables used for modelformulation, and may thus result in more robust tracking performance forvarying subjects. FIGS. 7, 8, 10, 11, and 12 , below, depict an exampleof capturing motion data with two different motion capture devicessimultaneously.

At step 104, one or more candidate inter-state differentiation variablesare determined based on the key state motion data captured in step 102.

In one embodiment, at each key state in the temporal sequence of statesassociated with a physical activity, a set of candidate inter-statedifferentiation variables can be generated from quantified variables,such as joint angles, body segment unit vector orientations, anddistances between specific joints, to name a few examples.

At step 106, one or more classifiers may be trained based on the one ormore candidate inter-state differentiation variables.

In one embodiment, classifiers may be trained to analyze all candidateinter-state differentiation variables generated in step 104. Approachesfor training the classifiers may include, for example, linear andnon-linear regression techniques, supervised machine learningtechniques, curve fitting, and clustering techniques, such as GaussianMixture Modeling (GMM) and k-means, to name a few.

During training of the classifiers, the values of each candidateinter-state differentiation variable at a given state may be compared toa combination of its values from all other states in the physicalactivity. In one embodiment, a classifier associated with eachindividual key state may be trained to produce a predictedclassification by comparing itself to the combination of all other keystates in the activity sequence. Thus, the classifiers may provide acontinuous quantitative measure of the level of classificationconfidence, which may be used to determine key state achievement duringtracking of a subject's motion.

In some embodiments, the statistical significance of each respectivecandidate inter-state differentiation variable is assessed using, forexample, classifier outcomes and information regarding the relativechange in the respective candidate inter-state differentiation variablevalue between states. In some embodiments, a subset of statecharacteristic metrics may be chosen (step 108) and retained in physicalactivity model 110 based on the significance analysis of eachinter-state differentiation variable. The reduction of inter-statedifferentiation variables based on the significance analysis maybeneficially reduce the computational resource needs associated with aphysical activity model and therefore enable the physical activity modelto run on lower power devices, such as portable electronic devices.However, the full set of inter-state differentiation variables may beused as state characteristic metrics if, for example, computationalefficiency is not a consideration and if each inter-statedifferentiation variable is a statistically significant discriminator ofdifferent states.

Note that while classifiers are described in this example, other typesof mathematical methods, such as those described above, can be utilizedand designed to perform similar functionality.

A result of classifier training in step 106 is the identification of oneor more state characteristic metrics at step 108. In this example, thestate characteristic metrics are configured to be used as part ofphysical activity model 110 in conjunction with the trained classifiersin order to identify key states in motion data, such as in step 114.Notably, identifying key states in motion data may be based on alikelihood or probability calculation that the key state exists in themotion data. In some cases, a likelihood measure may be compared to athreshold to determine whether or not the key state is in the motiondata. The description with respect to FIG. 4 , below, providesadditional detail regarding physical activity model generation.

Once physical activity model 110 is generated, such as by trainedclassifiers and state characteristic metrics, it may be used to trackanother subject's motion, to identify key states in that motion, and toquantify the performance of an observed physical activity (e.g., byreference to its key states) against an ideal physical activity statesequence, such as might be determined in steps 102-108. For example,physical activity model 110 may be used to monitor a patient 112undertaking physical therapy at home.

For example, subject 112 may perform a prescribed physical activity infront of a motion capture device so that motion capture data iscollected at step 114. The motion capture data is then used to calculatethe state characteristic metrics (defined in step 108) at step 116 andone or more state classification estimates are produced by physicalactivity model 110 at step 118 via the trained classifiers. In someembodiments, classifier outcomes may be a probabilistic valuerepresenting physical activity model 110's confidence that a subject(e.g., patient 112 in this example) is currently performing the keystate associated with each specific classifier.

The output of each classifier may be monitored throughout the continuousmotion of a physical activity (e.g., of patient 112) and the peakprobability values may be recorded. Further, as patient 112 movesthrough the sequence of states, the probability of each sequential stateis checked against a threshold value. If the peak probabilistic valuesof each state are above this threshold, then it may be determined thatthe subject has completed a successful repetition of the prescribedexercise, and a repetition tracker may be updated at step 120. In someembodiments, the threshold value is predetermined, while in others, itmay be dynamically computed. For example, the threshold value may bechanged over time as the number of repetitions increases, or as thetraining regimen proceeds. Further, the threshold value may responddynamically to the patient's performance so that it increases as apatient gets better or more consistent with its motions, oralternatively decreases if a patient is underperforming. These are justa few examples.

Notably, while exercises are used in the description of FIG. 1 andthroughout as one example context, the methods described herein areapplicable to any sort of physical activity. This includes biomechanicalanalysis of physical activity training, fitness regimens, and sportscience research, such as training proper form or detection of criticalmovements. The methods described herein also may be used for clinicalsciences, such as the analysis of posture, balance, gait, and motorcontrol. Further, the methods described herein can be used forgesture/pose recognition and detection in virtual reality, gamingapplications, robotics, manufacturing applications, and ergonomicstudies. Further yet, the state-identification models described hereincan also be applied in psychological studies for analysis on behavioraland physical response.

Flow 100 depicted in FIG. 1 presents many advantages over conventionalsystems and methods. For example, in flow 100, a specific set ofclassifiers, suited for a specific end task requirement, can be used formodeling activities without any change to the methodology presented. Forexample, if likelihood estimates of state classification are desired,statistical probabilistic models can be implemented to output a level ofconfidence while tracking an activity. Further, various classifiers canprovide additional information regarding repetition tracking and overallperformance of the subject.

Further, the physical activity models described herein can produce acontinuous valued output, which can be used to track progression betweenstates. This can provide subjects with live feedback regarding theiractivity progression and can be used as a guide to determine how farthey must move to achieve the desired state pose.

The physical activity model outcomes may further be used to provide aprediction of risk and allow for cautionary feedback to preventpotential injury during an activity. In this way, the live physicalactivity model feedback may reduce the risk of over extension ofmonitored body segments and joint angles. A subject may be more likelyto stop their motion once the intended state has been achieved and notover-exert themselves when live feedback is provided.

Further, the physical activity models described herein may be trainedusing a collection of data from varying subjects and varying devices toproduce more robust monitoring results.

The identification of state characteristic metrics, as described herein,can be automated based on the statistical significance of each candidateinter-state differentiation variable. This overcomes the laborious taskof manual identification of metrics in conventional methods.

Further, the physical activity models described herein can beautomatically formulated using statistically derived relationshipsbetween states. Conventional methods have relied on more heuristicapproaches utilizing discrete state-identification thresholds setmanually based on empirical observations of a model demonstrator. Theprobabilistic classifier approach described herein may further removepotential bias of the activity model creator and produce a morestatistically significant activity tracking methodology.

Further, physical activity models described herein benefit from theirdirect derivation from an actual full performance of a recordedactivity. Physical activity model generation from real activity issignificantly faster than heuristic approaches, which necessitate atrial and error approach during model development.

Further, the physical activity models described herein are easilytunable if slight revisions need to be made. For example, modelparameterization can be performed, and the model's classifier formulascan be scaled to track activities with varying ranges of motion usingthe same training dataset. Further, during model formulation, a user mayrefine the list of state characteristic metrics to place more emphasison tracking body segments of greater interest, such as depicted in FIG.9 . Further, as above, the threshold values of achieving key states maybe dynamically changed over time in order to increase the strictness ofexercise following as, for example, a rehabilitation regimen progresses,or as measured performance improves.

Though FIG. 1 depicts one example implementation, there are various waysof implementing the methodology described herein.

For example, a methodology in which a professional or other userexplicitly demonstrates each key state is not required. Alternativelykey states may be algorithmically determined from an analysis of themotion capture data.

As another example, physical activity models may also be used to detecterroneous positions that a subject should avoid during an activitysequence. In this regard, a physical activity model can be trained usingmotion data from a recorded erroneous pose, and formulated to identifywhen a subject has achieved such an undesirable position. These posesmay include positions that could potentially result in injury or areengaging incorrect body regions.

As another example, the physical activity models described herein can beused for patient screening. For example, patients with issues of limitedrange of motion or inflexibility of certain joints associated withcommon ailments may perform physical activities in a distinguishablematter. Identification models may be formulated based on training datafrom varying types of subjects to detect certain conditions a patientmay exhibit.

Further, the physical activity models described herein may be configuredto track activity progression and compliance throughout the course of anactivity set. A continuous valued model output may be used to determineif a subject is truly completing the full range of motion required for aprescribed physical activity. To this end, the subject's progressionover the course of the physical activity may be monitored and ultimatelyused to make alterations to the prescribed activity.

As described above, formulation of physical activity models maygenerally require quantified information regarding key states. This datamany come from varying sources or methods other than an optical-basedmotion capture source. A motion capture device may include opticalcamera systems with image processing, marker based tracking systems withvarious marker (active, passive, semi-passive, modulated) and detector(optical, radio frequency) types, depth camera systems with objectrecognition algorithms, inertial measurement units (IMUs), mechanicalexoskeleton motion capture systems, or magnetic flux measurement systemsor combinations of those system. Other potential sources include, butare not limited to, data extracted processing methods that comparesequential frames of data to determine differences. Depth cameras and/orpoint cloud mapping can also be used to extract information on varyingstates.

In some embodiments, physical activity models can be formulated fromdata expressed in various other coordinate spaces rather than thethree-dimensional (3D) space generally described herein. For example,two-dimensional (2D) data could be extracted from image processingtechniques, or 3D motion capture devices may project captured motiondata onto a 2D plane.

While the physical activity models may be time-invariant in the examplesdescribed herein, they are nevertheless fully capable of tracking andcomparing physical activity sequence timing. In some embodiments, idealtiming can be extracted from the training data to determine the desiredrate of transition between states. These transition periods betweenstates can be identified using the state classifiers and theirassociated probabilistic outputs. With discrete boundaries establishedbetween states and transition zones, time information can be recordedand binned into associated state/transition regions. Subject timingbetween-states, and during states, can be compared to that from theidealized motion recorded during model training.

Various statistical methods and classifiers can be applied forstate-differentiation. These include, but are not limited to, logisticregression, Gaussian mixture models, Bayes classifiers, k-meansclustering, artificial neural networks, decision tree classifiers,random forest regression, gradient tree boosting, and support vectormachines.

State detection, as described above, may be formulated by comparing eachstate to the combination of all other states in the activity sequence.Many other variations of state comparison are also effective forphysical activity model formulation. For example, multinomial regressiontechniques can be implemented for physical activity sequences with manystate-positions within the sequence. Inter-state differentiationvariables from each state can be compared to all other individualstates, thus producing classifiers comparing all possible two-itemcombinations of states. Outputs of these 1 vs. 1 type classifiers can becombined during state-tracking to produce an estimated classification.States may also be compared only to other states that are contiguous inthe activity sequence. Another effective method would be to compare allstates to solely the initial state, and the resulting model results canbe combined to produce state estimates during tracking.

Example Physical Activity State Sequence

As above, a physical activity may be defined as a sequence of key statesa subject performs throughout one full cycle of the physical activity.Expressing a full sequence of states in a physical activity using asubset of “key states” from the full sequence allows for atime-invariant definition of the physical activity, which is alsocomputationally more efficient. Given the kinematics of an articulatedhuman skeleton, an appropriately chosen subset does not reduce thefidelity or quality of the exercise definition.

FIG. 2 depicts example visualization from a motion capture systemshowing the difference between two physical activity states (poses inthis example) during a physical activity sequence that progresses fromState 1 to State 2 and back again to State 1. Notably, States 1 and 2may be key states in a physical activity sequence.

In this example, the subject is standing upright in State 1 and is thenextending one leg outward without a bend at the knee in State 2. Thus,in this example, it is evident that a subset of available body segments202 can be used to distinguish between States 1 and 2.

The difference between each identified body segment and joint betweenStates 1 and 2 may form a set of candidate inter-state differentiationvariables as described further with respect to FIG. 3 .

Example Candidate Inter-State Differentiation Variables

FIG. 3 depicts an example of determining candidate inter-statedifferentiation variables that may be used to distinguish between keystates of a physical activity sequence.

As depicted in FIG. 3 , it is possible to determine many candidateinter-state differentiation variables based on a relatively small numberof joints and body segments. For example, here candidate inter-statedifferentiation variables may include: the angle of joints 302 and 318from a reference, such as plane 316; the distance of joints 304, 306,322, and 326, from a reference, such as a point or plane 316; unitvector orientations (e.g., 310 and 314) associated with body segments,such as 308, 312, 320, and 324; and others.

Focusing on State 2, it is clear there exists a subset of the candidateinter-state differentiation variables most relevant to identifying State2 as compared to State 1. In this example, the angle of joints 302 fromreference plane 316; the distance of joints 304 and 306 from referenceplane 316; and unit vector orientations 310 and 314 associated with bodysegments 308 and 312 are most determinative of State 2 as they havechanged the most between State 1 and State 2.

Notably, tracking a subset of states (e.g., key states), such as State 1and State 2 in this example, rather than every possible state in aphysical activity motion sequence, allows for calibrating or tuning thestringency with which a particular physical activity requirement isfollowed. For example, here slight changes in body segments 320 and 324would not be considered when determining between State 1 and State 2.Beneficially, this allows subjects to deviate to a configurable degreefrom an ideal physical activity state sequence during transitionsbetween states if so desired. To refine tracking, additional key statescan be added to the activity to constrain the motion further.

Example Physical Activity Model Definition Based on Key States

FIG. 4 depicts an example flow 400 for defining a physical activitymodel based on a plurality of key states. In particular, in thisexample, the activity states 402 from FIGS. 2 and 3 are used as examplekey states for training physical activity model 408.

As discussed above with respect to FIG. 3 , a plurality of candidateinter-state differentiation variables 404 may be determined based ontrackable aspects identified in motion capture data, such as trackablebody segments and joint locations in States 1 and 2.

The candidate inter-state differentiation variables 404 may be analyzedat 406 using classification methods, such as those described above, toidentify a subset of inter-state differentiation variables that are mosteffective at distinguishing between States 1 and 2 (402). In thisexample, statistically significant inter-state differentiation variablesidentified for each key state are referred to as state characteristicmetrics, and finalized classifiers are formulated using the statemetrics associated with each key state, as shown with respect tophysical activity model 408.

In particular, in this example, candidate inter-state differentiationvariables 404 included variables {A, . . . , Z}, whereas the resultingstate characteristic metrics for State 1, based on the classifieranalysis at 406, are {A, C, R}. Similarly, the resulting statecharacteristic metrics for State 2, based on the classifier analysis at406, are {A, D, P}. As is the case in this example, selected inter-statedifferentiation variables may be included as state characteristicmetrics for more than one key state (here A is included in each), thoughin other examples, each key state may have a unique set of statecharacteristic metrics.

Further in this example, once the characteristic metrics have beendetermined, finalized versions of classifiers (e.g., State 1 Classifierand State 2 Classifier in physical activity model 408) are created tomonitor when each state has been achieved by the subject beingmonitored.

As mentioned above, training data regarding States 1 and 2 may becollected from sets of subjects performing the physical activity, andspecific training data may be identified (e.g., tagged) as associatedwith each key state. When the training data comes from a set ofsubjects, the candidate inter-state differentiation variables may becalculated for each subject, at each state, and then pooled together.Then, classifiers compare the inter-state differentiation variablevalues for the varying key states in the physical activity to determinewhich inter-state differentiation variables provide significantinformation for state detection and classification. Capturing trainingmotion data from varying subjects may improve the robustness of physicalactivity models such that they are capable of tracking a wider varietyof subjects, such as subjects of varying size and body composition.

Example Determination of Physical Activity State Probabilities Based onCaptured Motion Data

FIG. 5 depicts an example of determining state (e.g., key state)probabilities using a physical activity model 518 based on motion data,such as may be captured by a motion tracking system.

In the example depicted in FIG. 5 , a subject's current position 502 ismonitored by a motion tracking system. The motion data generated by themotion tracking system may include data regarding various trackedaspects of the subject's body, such as body segments and joints.

Motion data associated with specific state characteristic metrics areprovided to each state-specific set of state characteristic metrics,such as 504 and 506. The state characteristic metric data for each stateis then used by each state's classifier, such as 508 and 510, to produceprobabilistic classifier confidence outcomes for each state, such as 512and 514.

As the subject moves through a physical activity motion sequence,classifier outputs (e.g., state probabilities 512 and 514) are trackedand compared to state-achievement threshold values.

Further, in some embodiments, the order in which state poses areachieved may be monitored and compared to a sequence of key statesdefined by an ideal physical activity state sequence in order todetermine a number of successful repetitions of a physical activity. Anexample of this is described below with respect to FIG. 12 .

Example Activity State Achievement and Activity State SequenceRepetition Counting

FIG. 6 depicts an example of tracking a physical activity statesequence, such as an ideal physical activity state sequence.

Box 602 includes an indication 610 that a physical activity model hasdetermined that State 1 has been achieved based on, for example, one ormore state characteristic metrics and one or more state-specificclassifiers associated with State 1. Further in box 602, there is anindication 608 that the subject has now achieved the first state in adesired state sequence 612 that is defined in this example as aprogression from State 1 to State 2 and then back to State 1.

Similarly, as depicted in box 604, the physical activity model hasdetermined that State 2 has been achieved based on, for example, one ormore state characteristic metrics and one or more state-specificclassifiers associated with State 2. Further in box 604, there is anindication that the subject has now achieved the first state and thesecond state in the desired state sequence 612.

Finally, in box 606, the physical activity model has determined thatState 1 has been achieved once again. Further in box 606, there is anindication 614 that the subject has now achieved all of the states inthe desired state sequence 612, and that a successful repetition hasbeen counted.

Example Graphical User Interfaces for Recording Training Data for aPhysical Activity Model

FIG. 7 depicts an example user interface 700 for recording training datafor physical activity model development.

In particular, user interface 700 includes a portion 702 that shows livetracking data, which in this example includes captured motion data fromtwo motion tracking systems simultaneously. As described above,capturing motion data from multiple motion tracking systems may improvethe resulting physical activity model by providing more diverse trainingdata for classifier training (e.g., as in step 106 in FIG. 1 ). Further,motion data from different motion capture systems may be used togenerate physical activity models that are specific or optimized to themotion capture system.

User interface 700 further includes a plurality of mode selection tabs,including an exercise tracking mode tab 704, a model development tab706, and a data analysis tab 708. In this example, model development tab706 is selected and information regarding the physical activity isdisplayed. For example, here the name 710 of the physical activity beingmodeled is a “Hip Abduction Standing Left”, and this physical activityis being modeled with a key state sequence 712 of 1-2-1. Further, asdepicted in portion 702, a subject is currently demonstrating the State1 pose. In some embodiments, the first state (here, State 1) in adefined physical activity state sequence may be referred to as aninitial state.

FIG. 8 depicts another view of example user interface 700 in which asecond state (State 2) in the defined physical activity state sequenceis being demonstrated.

FIG. 9 depicts a portion 900 of user interface 700 (depicted in FIGS. 7and 8 ) in which various state characteristic metrics are displayed.

In particular, the set of state characteristics metrics 902 is displayedin order of statistical strength (e.g., significance) 904 foridentifying State 1. Further, a subset of the set of statecharacteristics is selected 906 for use in repetition tracking.

In some embodiments, the selection of state characteristic metrics maybe performed automatically initially based on criteria, such asindividual significance thresholds or cumulative significance for of themost significant state characteristic metrics. However, a user mayfurther modify the selection of the state characteristic metrics usingthis user interface.

Example Graphical User Interfaces for Tracking Motion Data Using aPhysical Activity Model

FIG. 10 depicts the graphical user interface 700 from FIGS. 7 and 8 inan activity tracking mode. The activity tracking mode is selected inthis example via mode tab 704.

As depicted, motion data regarding a subject is being captured by twodifferent motion tracking systems simultaneously. For each system, aphysical activity model is outputting state probabilities 1002A and1002B, which may also be referred to as classifier confidence levels,for all of the states defined as part of the physical activity beingmonitored. In this example, the physical activity being monitored is a“Hip Abduction Standing Left”, which is shown selected in user interfaceelement 1014. Notably, in this example, the probabilities 1002A and1002B are slightly different based on the two different motion trackingsystems. However, for both models, the probability of State 1 (as wasdefined in training in FIG. 7 ) being performed by the subject based onthe captured motion data is nearly 1, i.e., the physical activity modelis nearly certain that State 1 is being performed by the subject.

In this embodiment, user interface 700 includes a probability thresholdadjustment user interface element 1008, which sets the threshold abovewhich a state is determined based on the probability outputs 1002A and1002B. In this case, the threshold is set at 85%, which means the State1 probabilities of 0.998 pass the threshold test and the State 2probabilities of 0.003 and 0.002 do not pass the threshold test. In thisexample, each camera's state tracking probability is comparedindependently to the threshold, but in other embodiments, each camera'sstate tracking probability may be averaged for a single determination.

Further in this embodiment, user interface 700 includes a range ofmotion adjustment user interface element 1010, through which modelparameterization is performed to easily tune the model. By adjusting therange of motion, the model classifier formulas (e.g. regressioncoefficients) are scaled to track activities with varying ranges ofmotion, all from using the same training dataset.

Further in this embodiment, user interface 700 includes repetition countuser interface elements 1004 (one for each of the current motiontracking systems), which count the repetitions of the physical activity.In this case, because no repetitions have been completed, each count isat 0.

Further in this embodiment, user interface 700 includes a modelparameters user interface element 1006, which depicts characteristics ofthe current physical activity model, including the number of key states,the temporal sequence of the key states, and the state characteristicmetrics used to identify the key states in captured motion data.

FIG. 11 depicts another view of the graphical user interface 700 fromFIGS. 7 and 8 in an activity tracking mode.

As depicted, the subject has begun to transition from State 1 to State 2(as defined above in FIG. 8 ). Thus, the state classifier probabilities1002A and 1002B have changed in favor of State 2, but they have not yetexceeded the probability threshold of 0.85, as set by user interfaceelement 1008.

Further, progress indicator bars 1016A and 1016B indicate how close thesubject is to performing the current target state (State 2) with respectto the selected probability threshold, indicated at 1008.

FIG. 12 depicts another view of the graphical user interface 700 fromFIGS. 7 and 8 in an activity tracking mode.

As depicted, the subject has reached State 2 (as defined above in FIG. 8), and the state classifier probabilities 1002A and 1002B for State 2now exceed the probability threshold of 0.85, as set by user interfaceelement 1008. Consequently, the progress indicator bars 1016A and 1016Bhave now changed in appearance (in this example their color has changed)to indicate that State 2 has been reached based on the selectedprobability threshold.

Further, repetition counters 1004 now indicate one repetition has beencompleted because the subject has successfully performed the physicalactivity sequence from State 1 to State 2.

Example Method for Generating a Physical Activity Model

FIG. 13 depicts an example method 1300 for generating a physicalactivity model.

Method 1300 beings at step 1302 with receiving, via a motion capturedevice, motion data corresponding to a plurality of key statesassociated with a physical activity sequence. As above, motion data mayinclude static pose data as well as dynamic motion data, all captured bythe motion capture device.

Method 1300 then proceeds to step 1304 with determining a plurality ofjoint positions associated with each key state of the plurality of keystates, for example, as described above with respect to FIGS. 1 and 2 .

Method 1300 then proceeds to step 1306 with determining a plurality ofbody segment positions associated with each respective key state of theplurality of key states, for example, as described above with respect toFIGS. 1 and 2 .

Method 1300 then proceeds to step 1308 with determining a plurality ofinter-state differentiation variables associated with each key state ofthe plurality of key states, for example, as described above withrespect to FIGS. 1, 3, and 4 .

Method 1300 then proceeds to step 1310 with determining one or morestate characteristic metrics based on the plurality of inter-statedifferentiation variables for each key state of the plurality of keystates, for example, as described above with respect to FIGS. 1, 3 , and4.

Method 1300 then proceeds to step 1312 with determining a classifierbased on the one or more state characteristic metrics for each key stateof the plurality of key states, for example, as described above withrespect to FIGS. 1 and 4 .

The physical activity model can then be generated (alternatively,defined) based on the classifiers and state characteristics associatedwith each key state of the plurality of key states.

In some embodiments, method 1300 further comprises determining aplurality of joint angles associated with each key state of theplurality of key states, for example, as described above with respect toFIGS. 1 and 3 . In some embodiments, determining each of the pluralityof joint angles associated with a respective key state may be based onthe plurality of body segments positions associated with the respectivekey state. In some embodiments, determining the plurality of inter-statedifferentiation variables for the respective key state is further basedon the plurality of joint angles associated with the respective keystate of the plurality of key states.

In some embodiments of method 1300, determining one or more statecharacteristic metrics based on the plurality of inter-statedifferentiation variables comprises using one of: a machine-learningtechnique; a statistical method; or a pattern recognition approach.

In some embodiments of method 1300, the classifier for each respectivekey state is configured to provide a score indicating a likelihood ofthe respective key state in the received motion data.

In some embodiments of method 1300, determining the one or more statecharacteristic metrics for a respective key state further comprises:receiving a selection, via a user interface of an application, of one ormore state characteristic metrics.

In some embodiments of method 1300, determining the one or more statecharacteristic metrics for a respective key state further comprises:determining a statistical significance of each of the plurality ofinter-state differentiation variables for identifying the respective keystate; and selecting as the one or more state characteristic metrics asubset of the plurality of inter-state differentiation variables basedon the determined statistical significance of each of the plurality ofinter-state differentiation variables.

In some embodiments of method 1300, each inter-state differentiationvariable in the subset of the plurality of inter-state differentiationvariables has a statistical significance above a threshold value.

In some embodiments of method 1300, a sum of the statisticalsignificance value for each of the inter-state differentiation variablesin the subset of the plurality of inter-state differentiation variablesexceeds a threshold value.

In some embodiments of method 1300, each respective key state of theplurality of key states associated with the physical activity sequenceis defined by monitoring a subject performing the respective key state.

In some embodiments of method 1300, each respective key state of theplurality of key states associated with the physical activity sequenceis defined by an automated analysis of the received motion data.

In some embodiments of method 1300, the motion data comprises a firstsubset of motion data associated with a first subject and a secondsubset of motion data associated with a second subject.

In some embodiments of method 1300, the motion capture device comprisesa depth-sensing camera.

Example Method for Using a Physical Activity Model

FIG. 14 depicts an example method 1400 for using a physical activitymodel.

Method 1400 begins at step 1402 with receiving motion data from a motioncapture device.

Method 1400 then proceeds to step 1404 with providing the receivedmotion data to a physical activity model. In some embodiments of method1400, the physical activity model comprises: a plurality of classifiers,wherein each classifier of the plurality of classifiers is associatedwith a key state of a physical activity; and a plurality of statecharacteristic metrics, wherein each state characteristic metrics of theplurality of state characteristic metrics is associated with one or moreof the plurality of classifiers.

Method 1400 then proceeds to step 1406 with receiving, from the physicalactivity model, a plurality of scores. In some embodiments of method1400, each score of the plurality of scores is associated with one ofthe plurality of classifiers.

Method 1400 then proceeds to step 1408 with determining a key state isrepresented in the received motion data based on the plurality ofscores.

In some embodiments of method 1400, determining the key state isrepresented in the received motion data based on the plurality of scoresfurther comprises: determining a score associated with the key stateexceeds a threshold value.

In some embodiments of method 1400, the score associated with the keystate indicates a probability that the key state is represented in thereceived motion data.

In some embodiments, method 1400 further comprises displaying theplurality of scores in a graphical user interface on a display device.

In some embodiments, method 1400 further comprises indicating within thegraphical user interface on the display device when a respective scoreof the plurality of scores exceeds a threshold by changing an attributeof the respective score in the graphical user interface, wherein theattribute comprises one or more of: a color of the respective score, asize of the respective score, or a format of the respective score.

In some embodiments, method 1400 further comprises incrementing arepetition count for the physical activity based on a sequence ofreceived scores; and displaying the repetition count in a graphical userinterface on a display device.

In some embodiments of method 1400, the motion capture device is adepth-sensing camera.

Example Processing System

FIG. 15 depicts an example processing system 1500 configured to generateand use physical activity models.

For example, processing system 1500 may be configured to perform one ormore aspects of flow 100 described with respect to FIG. 1 , flow 400described with respect to FIG. 4 , and methods 1300 and 1400 describedwith respect to FIGS. 13 and 14 , respectively.

Processing system 1500 includes a CPU 1502 connected to a data bus 1550.CPU 1502 is configured to process computer-executable instructions,e.g., stored in memory 1510 or storage 1530, and to cause processingsystem 1500 to perform methods as described herein. CPU 1502 is includedto be representative of a single CPU, multiple CPUs, a single CPU havingmultiple processing cores, and other forms of processing architecturecapable of executing computer-executable instructions.

Processing system 1500 further includes input/output device(s) 1504,which may include motion capture or tracking devices as describedherein, as well as input/output interface(s) 1506, which allowprocessing system 1500 to interface with input/output devices, such as,for example, keyboards, displays, mouse devices, pen input, motioncapture or tracking devices, motion tracking sensors, and other devicesthat allow for interaction with processing system 1500.

Processing system 1500 further includes network interface 1508, whichprovides processing system 1500 with access to external networks, suchas network 1514.

Processing system 1500 further includes memory 1510, which in thisexample includes a plurality of components.

For example, memory 1510 includes deep neural receiving component 1512,which is configured to perform receiving functions as described above,for example, with respect to methods 1300 and 1400.

Memory 1510 further includes determining component 1514, which isconfigured to perform determining functions as described above, forexample, with respect to methods 1300 and 1400.

Memory 1510 further includes defining component 1516, which isconfigured to perform defining functions as described above, forexample, with respect to methods 1300 and 1400.

Memory 1510 further includes selecting component 1518, which isconfigured to perform selecting functions as described above, forexample, with respect to methods 1300 and 1400.

Memory 1510 further includes providing component 1520, which isconfigured to perform providing functions as described above, forexample, with respect to methods 1300 and 1400.

Memory 1510 further includes displaying component 1522, which isconfigured to perform selecting functions as described above, forexample, with respect to methods 1300 and 1400.

Note that while shown as a single memory 1510 in FIG. 15 for simplicity,the various aspects stored in memory 1510 may be stored in differentphysical memories, but all accessible CPU 1502 via internal dataconnections, such as bus 1550.

Processing system 1500 further includes storage 1530, which in thisexample includes training data 1532 (e.g., motion capture data fortraining a physical activity model), live data 1534 (e.g., live motioncapture data provided to a physical activity model), statecharacteristic metrics 1536, classifiers 1538, and physical activitymodels 1540. Note that while shown as separate items for clarity, insome embodiments, a physical activity model comprises a collection ofclassifiers and state characteristics metrics.

While not depicted in FIG. 15 , other aspects may be included in storage1510.

As with memory 1510, a single storage 1530 is depicted in FIG. 15 forsimplicity, but the various aspects stored in storage 1530 may be storedin different physical storages, but all accessible to CPU 1502 viainternal data connections, such as bus 1550, or external connection,such as network interface 1508.

Notably, while shown as a single processing system in the exampledepicted in FIG. 15 , other embodiments may include decoupled portionsthat function together as a processing system. For example, the variouscomponents in memory 1510 and data in storage 1530 may be implemented orstored across a network of processing systems, or in a cloud-basedprocessing system, or in combinations of the same. For example, in someembodiments, training data 1532, physical activity models 1540,classifiers 1538, and state characteristics 1536 may be stored remotefrom a motion tracking system that captures live data 1534.

For example, a patient may have a client processing system that includesa motion tracking I/O device that captures lives data and feeds it backto a server processing system. Similarly, the patient's clientprocessing system may store physical activity models 1540, classifiers1538, and state characteristics 1536 locally, which were generatedremotely, and which were downloaded to the local client processingsystem over a network connection, such as the Internet.

Further, processing system 1500 may be configured to function as atraining system or a tracking system. Other embodiments of processingsystems may be a training system only, or a tracking system only. Forexample, patients may receive only tracking systems.

In general, processing system 1500 is just one possible embodiment, andthe various aspects of processing system 1500 may be distributed acrossa plurality of devices, may be omitted, or added as necessary for any ofthe particular functions or methods described herein.

Example Embodiments

Clause 1: A method of generating a physical activity model, comprising:receiving, via a motion capture device, motion data corresponding to aplurality of key states associated with a physical activity sequence;for each respective key state in the plurality of key states:determining a plurality of joint positions associated with therespective key state; determining a plurality of body segment positionsassociated with the respective key state based on the plurality of jointpositions associated with the respective key state; and determining aplurality of inter-state differentiation variables for the respectivekey state based on one or more of: the plurality of joint positionsassociated with the respective key state; or the plurality of bodysegment positions associated with the respective key state; determiningone or more state characteristic metrics for the respective key statebased on the plurality of inter-state differentiation variablesassociated with the respective key state; and determining a classifierfor the respective key state based on the one or more statecharacteristic metrics; and defining a physical activity model based onthe one or more state characteristic metrics and the classifierassociated with each key state.

Clause 2: The method of Clause 1, wherein the classifier for therespective key state is configured to provide a score indicating alikelihood of the respective key state in the received motion data.

Clause 3: The method of Clause 1 or 2, wherein determining the one ormore state characteristic metrics for the respective key state furthercomprises: receiving a selection, via a user interface of anapplication, of one or more state characteristic metrics.

Clause 4: The method of any of Clauses 1-3, wherein determining the oneor more state characteristic metrics for the respective key statefurther comprises: determining a statistical significance of each of theplurality of inter-state differentiation variables for identifying therespective key state; and selecting as the one or more statecharacteristic metrics a subset of the plurality of inter-statedifferentiation variables based on the determined statisticalsignificance of each of the plurality of inter-state differentiationvariables.

Clause 5: The method of Clause 4, wherein each inter-statedifferentiation variable in the subset of the plurality of inter-statedifferentiation variables has a statistical significance above athreshold value.

Clause 6: The method of Clause 4, wherein a sum of the statisticalsignificance value for each of the inter-state differentiation variablesin the subset of the plurality of inter-state differentiation variablesexceeds a threshold value.

Clause 7: The method of any of Clauses 1-6, further comprising:determining a plurality of joint angles associated with the respectivekey state based on the plurality of body segments positions associatedwith the respective key state, wherein determining the plurality ofinter-state differentiation variables for the respective key state isfurther based on the plurality of joint angles associated with therespective key state of the plurality of key states.

Clause 8: The method of any of Clauses 1-7, wherein each respective keystate of the plurality of key states associated with the physicalactivity sequence is defined by monitoring a subject performing therespective key state.

Clause 9: The method of any of Clause 1-8, wherein the motion datacomprises a first subset of motion data associated with a first subjectand a second subset of motion data associated with a second subject.

Clause 10: The method of any of Clauses 1-9, wherein the motion capturedevice comprises a depth-sensing camera.

Clause 11: A processing system, comprising: a non-transitorycomputer-readable medium comprising computer-executable instructions;and a processor configured to execute the computer-executableinstructions and cause the processing system to perform a method ofgenerating a physical activity model, the method comprising: receiving,via a motion capture device, motion data corresponding to a plurality ofkey states associated with a physical activity sequence; for eachrespective key state in the plurality of key states: determining aplurality of joint positions associated with the respective key state;determining a plurality of body segment positions associated with therespective key state based on the plurality of joint positionsassociated with the respective key state; and determining a plurality ofjoint angles associated with the respective key state based on theplurality of body segments positions associated with the respective keystate; determining a plurality of inter-state differentiation variablesfor the respective key state based on one or more of: the plurality ofjoint positions associated with the respective key state; or theplurality of body segment positions associated with the respective keystate; determining one or more state characteristic metrics for therespective key state based on the plurality of inter-statedifferentiation variables associated with the respective key state; anddetermining a classifier for the respective key state based on the oneor more state characteristic metrics; and defining a physical activitymodel based on the one or more state characteristic metrics and theclassifier associated with each key state.

Clause 12: The processing system of Clause 11, wherein the classifierfor the respective key state is configured to provide a score indicatinga likelihood of the respective key state in the received motion data.

Clause 13: The processing system of any of Clauses 11 or 12, whereindetermining the one or more state characteristic metrics for therespective key state further comprises: receiving a selection, via auser interface of an application, of one or more state characteristicmetrics.

Clause 14: The processing system of any of Clauses 11-13, wherein themethod further comprises: determining a plurality of joint anglesassociated with the respective key state based on the plurality of bodysegments positions associated with the respective key state, whereindetermining the plurality of inter-state differentiation variables forthe respective key state is further based on the plurality of jointangles associated with the respective key state of the plurality of keystates.

Clause 15: A method for using a physical activity model, comprising:receiving motion data from a motion capture device; providing thereceived motion data to a physical activity model, wherein the physicalactivity model comprises: a plurality of classifiers, wherein eachclassifier of the plurality of classifiers is associated with a keystate of a physical activity; and a plurality of state characteristicmetrics, wherein each state characteristic metrics of the plurality ofstate characteristic metrics is associated with one or more of theplurality of classifiers; receiving, from the physical activity model, aplurality of scores, wherein each score of the plurality of scores isassociated with one of the plurality of classifiers; and determining akey state is represented in the received motion data based on theplurality of scores.

Clause 16: The method of Clause 15, wherein determining the key state isrepresented in the received motion data based on the plurality of scoresfurther comprises: determining a score associated with the key stateexceeds a threshold value.

Clause 17: The method of Clause 16, wherein the score associated withthe key state indicates a probability that the key state is representedin the received motion data.

Clause 18: The method of any of Clauses 15-17, further comprising:displaying the plurality of scores in a graphical user interface on adisplay device.

Clause 19: The method of Clause 18, further comprising: indicatingwithin the graphical user interface on the display device when arespective score of the plurality of scores exceeds a threshold bychanging an attribute of the respective score in the graphical userinterface, wherein the attribute comprises one or more of: a color ofthe respective score, a size of the respective score, or a format of therespective score.

Clause 20: The method of any of Clauses 15-19, further comprising:incrementing a repetition count for the physical activity based on asequence of received scores; and displaying the repetition count in agraphical user interface on a display device.

Clause 21: The method of any of Clauses 15-20, wherein the motioncapture device is a depth-sensing camera.

The preceding description is provided to enable any person skilled inthe art to practice the various embodiments described herein. Theexamples discussed herein are not limiting of the scope, applicability,or embodiments set forth in the claims. Various modifications to theseembodiments will be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherembodiments. For example, changes may be made in the function andarrangement of elements discussed without departing from the scope ofthe disclosure. Various examples may omit, substitute, or add variousprocedures or components as appropriate. For instance, the methodsdescribed may be performed in an order different from that described,and various steps may be added, omitted, or combined. Also, featuresdescribed with respect to some examples may be combined in some otherexamples. For example, an apparatus may be implemented or a method maybe practiced using any number of the aspects set forth herein. Inaddition, the scope of the disclosure is intended to cover such anapparatus or method that is practiced using other structure,functionality, or structure and functionality in addition to, or otherthan, the various aspects of the disclosure set forth herein. It shouldbe understood that any aspect of the disclosure disclosed herein may beembodied by one or more elements of a claim.

As used herein, the word “exemplary” means “serving as an example,instance, or illustration.” Any aspect described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother aspects.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover a, b, c,a-b, a-c, b-c, and a-b-c, as well as any combination with multiples ofthe same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b,b-b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Also, “determining” may include receiving (e.g., receiving information),accessing (e.g., accessing data in a memory) and the like. Also,“determining” may include resolving, selecting, choosing, establishingand the like.

The methods disclosed herein comprise one or more steps or actions forachieving the methods. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims. Further, thevarious operations of methods described above may be performed by anysuitable means capable of performing the corresponding functions. Themeans may include various hardware and/or software component(s) and/ormodule(s), including, but not limited to a circuit, an applicationspecific integrated circuit (ASIC), or processor. Generally, where thereare operations illustrated in figures, those operations may havecorresponding counterpart means-plus-function components with similarnumbering.

The following claims are not intended to be limited to the embodimentsshown herein, but are to be accorded the full scope consistent with thelanguage of the claims. Within a claim, reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more.” Unless specifically statedotherwise, the term “some” refers to one or more. No claim element is tobe construed under the provisions of 35 U.S.C. § 112(f) unless theelement is expressly recited using the phrase “means for” or, in thecase of a method claim, the element is recited using the phrase “stepfor.” All structural and functional equivalents to the elements of thevarious aspects described throughout this disclosure that are known orlater come to be known to those of ordinary skill in the art areexpressly incorporated herein by reference and are intended to beencompassed by the claims. Moreover, nothing disclosed herein isintended to be dedicated to the public regardless of whether suchdisclosure is explicitly recited in the claims.

The invention claimed is:
 1. An apparatus, comprising: a processor; anda memory coupled to the processor, the memory comprising instructionsthat, when executed by the processor, cause the processor to: accesstraining motion data of a physical activity sequence, determine asequence of a plurality of key states in the training motion data, eachof the plurality of key states comprising a specific state that in-partdefines the physical activity sequence, determine at least one statecharacteristic metric for each of the plurality of key states, the atleast one state characteristic metric comprising at least one variableconfigured to identify a state of the plurality of key states, anddetermine a physical activity model based on the at least one statecharacteristic metric associated with each of the plurality of keystates.
 2. The apparatus of claim 1, the physical activity sequencecomprising an exercise performed by a human body.
 3. The apparatus ofclaim 1, the instructions, when executed by the processor, to cause theprocessor to, for each of the plurality of key states, determine atleast one body segment position associated with at least one jointposition.
 4. The apparatus of claim 3, the instructions, when executedby the processor, to cause the processor to, for each of the pluralityof key states, determine a plurality of inter-state differentiationvariables comprising variables used to differentiate between theplurality of key states.
 5. The apparatus of claim 4, the plurality ofinter-state differentiation variables determined based on one or more ofthe at least one body segment position or the at least one jointposition.
 6. The apparatus of claim 4, the at least one statecharacteristic metric comprising a subset of the plurality ofinter-state differentiation variables used to identify a state of theplurality of key states.
 7. The apparatus of claim 6, wherein the atleast one inter-state differentiation variable comprises a plurality ofinter-state differentiation variables, the instructions, when executedby the processor, to cause the processor to: determine a statisticalsignificance of each of the plurality of inter-state differentiationvariables for identifying a respective state of the plurality of keystates, and determine the at least one characteristic metric as a subsetof the plurality of inter-state differentiation variables based on thestatistical significance.
 8. The apparatus of claim 1, the instructions,when executed by the processor, to cause the processor to, for each ofthe plurality of key states, determine a classifier based on the atleast one state characteristic metric, the classifier configured todetermine a probability that one of the plurality of key states is incaptured motion data.
 9. The apparatus of claim 8, the physical activitymodel based on the at least one state characteristic metric and theclassifier associated with each of the plurality of key states.
 10. Theapparatus of claim 1, the plurality of key states determined by at leastone of: user input via a graphical user interface selecting at least oneof the plurality of key states, or automatically based on analysis ofthe training motion data based on indicators of a transition between atleast two of the plurality of key states.
 11. A method, comprising, viaa processor of a computing device: accessing training motion data of aphysical activity sequence; determining a sequence of a plurality of keystates in the training motion data, each of the plurality of key statescomprising a specific state that in-part defines the physical activitysequence; determining at least one state characteristic metric for eachof the plurality of key states, the at least one state characteristicmetric comprising at least one variable configured to identify a stateof the plurality of key states; and determining a physical activitymodel based on the at least one state characteristic metric associatedwith each of the plurality of key states.
 12. The method of claim 11,the physical activity sequence comprising an exercise performed by ahuman body.
 13. The method of claim 11, comprising determining at leastone body segment position associated with at least one joint position.14. The method of claim 13, comprising determining a plurality ofinter-state differentiation variables comprising variables used todifferentiate between the plurality of key states.
 15. The method ofclaim 14, the plurality of inter-state differentiation variablesdetermined based on one or more of the at least one body segmentposition or the at least one joint position.
 16. The method of claim 14,the at least one state characteristic metric comprising a subset of theplurality of inter-state differentiation variables used to identify astate of the plurality of key states.
 17. The method of claim 16,wherein the at least one inter-state differentiation variable comprisesa plurality of inter-state differentiation variables, comprising:determining a statistical significance of each of the plurality ofinter-state differentiation variables for identifying a respective stateof the plurality of key states; and determining the at least onecharacteristic metric as a subset of the plurality of inter-statedifferentiation variables based on the statistical significance.
 18. Themethod of claim 11, comprising determining a classifier based on the atleast one state characteristic metric, the classifier configured todetermine a probability that one of the plurality of key states is incaptured motion data.
 19. The method of claim 18, the physical activitymodel based on the at least one state characteristic metric and theclassifier associated with each of the plurality of key states.
 20. Themethod of claim 11, the plurality of key states determined by at leastone of: user input via a graphical user interface selecting at least oneof the plurality of key states, or automatically based on analysis ofthe training motion data based on indicators of a transition between atleast two of the plurality of key states.